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Chapter 1 


Blended Learning: 
Choosing a Recipe 
for Success 


In the 2017-18 school year, New Leaders studied a series 

of innovative programs for public K-12 school leaders and 
faculty that incorporated independent online instructional 
components with in-person professional development 
sessions, job-embedded practice, and personalized coaching. 


New Leaders positioned itself to leverage the use of technology in delivering 
high-quality adult learning to a wide span of geographic regions, and in accom- 
modating the scheduling needs of busy education professionals who are unable 
to attend frequent in-person sessions. In adopting a blended learning paradigm 
for professional development, New Leaders paved the way for nimbler, more 
integrated, and, ultimately, more effective partnerships. 


For the past 17 years New Leaders has provided crucial, high-quality professional 
development that fosters leadership skills in working educators. Multiple independent 
studies have confirmed the effectiveness of New Leaders’ training for improving student 
achievement and school climate." 


New Leaders is committed to using data to continuously improve its programming and 
partnerships. With a longstanding history of sharing lessons with the field to advance 
effective leadership practices, the organization sought to document and examine the 
development and implementation of its new blended learning programs. The goals were 
to learn what worked and what didn’t — and to make specific improvements that could 
yield stronger results for participants and their students in the future. 


Two decades of peer-reviewed studies of in-person, online, and blended instruction have 
consistently demonstrated that quality trumps delivery method. High-quality blended 
programs outperform low-quality online or in-person programs, and vice versa.? 


1 Gates, S. M., Hamilton, L. S., Martorell, P., Burkhauser, S., Heaton, P., Pierson, A.,... Gu, K. (2014). Preparing principals to raise student 
achievement: Implementation and effects of the New Leaders program in ten districts — Appendix. Santa Monica, CA: RAND Corporation. 

2 Gates, S.M., Baird, M.D., Doss, C.J., Hamilton, L.S., Opper, I.M., Master, B.K., ...Zaber, M.A. (2019). Preparing school leaders for success: Evaluation 
of New Leaders’ Aspiring Principals Program, 2012-2017. Santa Monica, CA: RAND Corporation. 


3 Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and 
review of online learning studies. Washington, DC: U.S. Department of Education. Retrieved from http://www.ed.gov/about/offices/list/opepd/ 
ppss/reports.html 
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Although no published research currently addresses the delivery of blended leadership 
development programs to teams of in-service teachers and administrators,* examples from 
higher education and adult learning suggest that well-supported blended learning can be 
effective in this context. 


Drawing on blended and adult learning theories, this report presents the findings of New 
Leaders’ school year 2017-18 mixed-methods investigation into its blended learning programs. 
A comparative analysis of programs in districts and charter networks in five different regions 
across the United States reveals the opportunities and challenges of these programs. 


Developing and implementing strong blended adult learning engagements has much in 
common with adapting tried-and-true recipes and creating new ones to use with new 
kitchen tools: Successful experimentation depends on knowledge of the ingredients 
and tools, and on an explicit understanding of the desired outcome. Just as there are 
many ways to make a delicious lasagna, there are many ways to build and deliver 
effective blended adult instruction — provided that the essentials are 

in place and that the cooks know what to do with them. This investigation of 

New Leaders’ blended engagements in school year 2017-18 reveals what those 
essentials are and offers suggestions for tailoring the necessary components 

of blended adult learning to the local context. 


Methodology 


This study addressed the following questions: 
1. What are the features of effective blended learning models for adult professional development? 
2. What lessons for the field of adult professional development do New Leaders’ blended program models offer? 


3. What are the successes, opportunities, and challenges that New Leaders has encountered in developing and 
delivering blended programs? 


To address these questions, New Leaders conducted a series of program observations and 
interviews with blended-program participants. Two New Leaders researchers spoke with 
participants in five New Leaders’ blended learning programs during the 2017-18 school year. 
Interviews with program participants were conducted in-person and were recorded.* 
District names have been removed to retain confidentiality. 


Coded interview transcripts captured the catalysts and dampeners (things that either help 

or hinder) of program completion. In participant interviews, the team identified five thematic 
categories that closely align with the literature on blended learning (see Appendix B). The team 
applied the coding scheme to the interview transcripts, and double-coded a subset to ensure 
the reliability of the analysis.** 


This study also includes: interview data from New Leaders staff who were directly involved 

in scoping, designing, and implementing blended learning engagements during school year 
2017-18; 2017-18 participant survey data routinely collected by New Leaders; and user meta- 
data from Canvas, a learning management system that provides insight into how participants 
used the online resources New Leaders made available to them. 


4 Voci, E., & Young, K. (2001). Blended learning working in a leadership development programme. Industrial and Commercial Training, 33(5), 157-161. 
Interview participants were compensated for their time with DonorsChoose gift cards. 


(Cohen's kappa = .72) 
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Chapter 2 


Prepping for Blended Learning: 
Organizational Alignment, 
Resource Allocation, Staff 
Buy-In, and Key Ingredients 


Many institutions see blended learning as a panacea that promises less overhead and 
better learning outcomes.® However, research shows that high-quality blended learning 
must begin with an organization's commitment both to garner the staff's support for 
this complex endeavor and to provide adequate start-up resources — technological 
resources, staff time, and financial support.®” 


Whether for school-age students or adult learners, blended learning programs 
may initially be more expensive than traditional in-person programs because 
of essential investments in technology and other areas of support.®? Orga- 
nizations that implement blended instruction must first precisely map out 

the necessary resources — in technology, instructional staff, and support 

staff — and must make sure that organizational structures are in place to 
support the transition. 


|. Organizational Alignment: Using a New Tool ina 
New Context — New Leaders’ Flexible Engagements 


Some of the blended engagements in this study were new partnerships in school 
year 2017-18, whereby New Leaders collaborated closely with districts on targeted 
programming to address their most pressing priorities. New Leaders and its partners 
worked together to select “a la carte” from a menu of research-based instructional 
content drawn from established New Leaders programs. 


These “flexible” engagements could differ from established New Leaders programs in 
several ways: duration (e.g., four or six months, instead of a whole school year); content 
(e.g., mixing content from different established New Leaders programs); or designated 
participants (e.g., principals and their leadership teams alongside principal supervisors). 


5 Garrison, D. R., & Vaughan, N. D. (2008). Blended learning in higher education: Framework, principles, and guidelines. San Francisco: John 
Wiley & Sons. 


6 Moskal, P., Dziuban, C., & Hartman, J. (2013). Blended learning: A dangerous idea? The Internet and Higher Education, 18, 15-23. 


7 Stacey, E., & Gerbic, P. (2008). Success factors for blended learning. Hello! Where are you in the landscape of educational technology? 
Proceedings ASCILITE Melbourne 2008, 964-968. 


8 Means, B. et al. (2009). 


9 Brown, J., & Kurzweil, M. (2017). Instructional Quality, Student Outcomes, and Institutional Finances. Washington, DC: American Council on 
Education. Retrieved from https:/Awww.acenet.edu/news-room/Pages/Instructional-Quality-Student-Outcomes-and-Institutional-Fi- 
nances.aspx 
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As it embarked on delivering blended content in new ways during school year 2017-18, New 
Leaders was also finding its feet with new, flexible program models. Some staff members felt 
underprepared to navigate new engagements that were both blended and flexible. 


Il. Adequate Resource Allocation: The Costs of Blended Learning 


Building blended instructional programs can be expensive. The prevailing assumption that 
blended programming is cheaper to produce and deliver is widespread; when unexamined, 
this belief can undermine efforts to set up high-quality blended programming."° If budgets 
are set without accounting for the extra staff hours needed for training and for providing 
ongoing technical support to staff and participants, essential resources may not be in place 
from the start; furthermore, staff may miss or de-prioritize those needs once the program 
is underway (especially if they are competing with other urgent needs for finite financial or 
time resources). Researchers struggle to calculate the relative costs of online, blended, and 
in-person programs because of vast differences among programs, institutional accounting 
practices, the cost formulas used for comparison, and other issues. For example, when one 
widely reported study showed that blended learning at the University of Central Florida 
reduced overall expenditures by nearly 40 percent, critics pointed out that this figure omitted 
many start-up costs and ongoing supports involved in the virtual program components, 
because the calculations focused primarily on instructor time and facilities costs." 


New Leaders developed virtual content starting in 2014, with generous support from a host 
of foundations.” The content initially served as a standalone, virtual program; it later evolved 
into discrete modules for blended engagements. Our experience underscores the start-up 
costs associated with creating content on the front end, and then providing adequate 
ongoing support for instructors and students. 


Ill. Buy-In from Staff and Cross-Team Coordination 


While recognizing that blended instructional models have great “Initially, | spent a lot of time with the 
potential, research shows that teaching fully or partially online has platform. When I'm in the room, it's 


required more of instructors’ time than teaching in person — as Soa fei sti oa dibei 
much as three times more.” These issues of skill and time converge aDOME THY TACISRIOSty re ae OCI. 


as instructors often find themselves providing front-line technical able to read the room, faces, etc. When 
support to students, a time-consuming task for which they may not be you're online, you have to anticipate 
prepared. But when adequately supported, blended instructors often and solve for technology problems 


j H j j it] j 14 
realize greater student satisfaction than in traditional course settings. before the program actually starts.” 


Among other decisions, designers of blended adult professional NEW LEADERS PROGRAM IMPLEMENTATION STAFF 
development programming must determine how to create a 

thriving in-person and online community, engage participants, 

build in flexibility, and facilitate participant learning." 


0 Bakia, M., Shear, L., Toyama, Y., & Lasseter, A. (2012). Understanding the Implications of Online Learning for Educational Productivity. Washington, 
DC: U.S. Department of Education, Office of Educational Technology. 


1 Bakia et al. (2012). 

2 Generous support for New Leaders’ development of online content was provided by Charles and Helen Schwab, the Wasserman Foundation, 
and Anonymous. 

3 Ocak, M. A. (2010). Blend or not to blend: A study investigating faculty members’ perceptions of blended teaching. World Journal on 
Educational Technology, 2(3), 196-210. 

4 Caravias, V. (2018). Teachers’ conceptions and approaches to blended learning: A literature review. In Oncioiu, |. (Ed.), Driving Innovation and 
Business Success in the Digital Economy, pp. 228-251. Hershey, PA: IG! Global. 


5 Boelens, R., De Wever, B., & Voet, M. (2017). Four key challenges to the design of blended learning: A systematic literature review. Educational 
Research Review, 22, 1-18. 
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New Leaders staff are keenly aware of how highly participants value opportunities to 
network with their peers and interact personally with their instructors: 


[Participants] seem to be most interested in the videos, exemplars of assignments, things that they 
can read through and watch at their own pace to prepare for the in-person sessions, so that they can 
really use the session time to connect with their peers that they otherwise don’t usually see.” 

NEW LEADERS PROGRAM IMPLEMENTATION STAFF 


When creating their own blended courses, many online instructors have discovered that a 
high-quality educational experience requires more than retrofitting an existing in-person 
course with online discussion boards and YouTube links. The design and delivery of online 
blended courses — for children or adults — require a dramatically different technological 
and pedagogical skill set than the design and delivery of courses taught in-person 
demand.'® The best of both worlds, as blended learning is sometimes characterized, 

is often achieved when a team builds new courses from scratch. 


An effective blended learning program design is not simply a ratio of time spent in-person 
versus online.” Surveys of adult learners reveal that they are eager for more experimental 
and integrative approaches to blended instruction, even as instructors and institutions 
favor more conservative, traditional-looking programs that are perceived to be less likely 
to fail in engaging students and imparting learning.’® Perhaps unsurprisingly, surveys 
show that teachers are more willing than other professionals to experiment with different 
ways of learning that use online resources, and have provided useful feedback to course 
designers to shape educational programs for adult learners.” 


“| heard...that something new was coming down the pipe to build leadership teams, and that we 
should be on the lookout for it, and it might be something that we should consider as principals to 
help move the work inside the schools. So, once it did come down, | jumped on board immediately... 
Land at the end of that program] said, ‘Hey, if you guys are having it for ‘17 or ‘18, my school needs 
to be a part of it.” 

"V," PARTICIPATING PRINCIPAL 


“It was kind of amazing — even though [our principal] wasn’t with us for that first video watching, 
we came away with the same ideas. We can see our way of work, and then we can also be very 
reflective on our individual styles, our individual way of work. The last [video] we did, we all sat and 
did it together and, again, we were able to talk [about]...our strengths as well as our weaknesses 
and know how to be better as a team. [The videos] helped us develop a way of work that supports 
the bigger picture.” 

"N," PARTICIPATING TEACHER 


16 Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher education. The Internet and 
Higher Education, 7(2), 95-105. 


17 Graham C. (2013). Emerging practice and research in blended learning. In Handbook of Distance Education (3rd edition), Michael Grahame 
Moore (Ed.), p. 334. New York: Routledge. 


18 Mirriahi, N., & Alonzo, D. (2015). Shedding light on students’ technology preferences: Implications for academic development. Journal of 
University Teaching & Learning Practice, 12(1), 6. 


19 Koutsodimou, K., & Jimoyiannis, A. (2015). MOOCs for teacher professional development: Investigating views and perceptions of the 
participants. In Proceedings of the 8th International Conference of Education, Research and Innovation, 18-20 November. 
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IV. Five Key Ingredients of Blended 
Learning for Adult Audiences 


Our research identified five prevalent themes in blended 
participants’ accounts of their experiences with New Leaders’ 
programs: Accountability, Prior Experience, Motivation, 
Self-Regulation, and Relevance. These themes are not new; 
decades of research in adult learning previously revealed the 
importance of similar principles when teaching adults. In general, best 

practices in adult blended learning mirror strong in-person learning experiences — albeit 
redefined, tweaked, and combined in interesting ways to accommodate participants’ use of 
technology in the learning process. 


The five themes outlined below describe aspects of participants’ experiences in New 
Leaders’ programs that are distinct but often interdependent. For example, a participant who 
values the relevance of the program's content to her work might recognize that in order to 
maximize her learning she will need to deploy self-regulation strategies to manage her time 
and engagement effectively. Likewise, participants who are motivated by the opportunities to 
engage with their peers in rich discussions of content assigned as pre-work can find group 
sessions to be an effective accountability mechanism, which in turn encourages them to use 
self-regulation strategies to complete assignments in advance. 


(1.) Accountability 


“Now we are being held accountable for [our learning in the program] because whatever work we 
bring back [to our teams], we created it, where before, we'd read this and report out on it. That's 
completely different from actually going and seeing how is that going to look.” 

“C," PARTICIPATING TEACHER 


One tenet in teaching adult learners is to allow them to assume responsibility for their own 
learning. Of special significance in adult professional development is how individuals see 
their efforts contributing to the benefit of the group's goals. In online and blended learning 
environments, this type of cooperation can be cultivated by learners as they share goals, 
resources, roles, tasks, and accomplishments.° 


When participants in New Leaders’ blended programs felt that their “The session plans assume that the 
supervisors or districts engaged with and supported New Leaders participants are already familiar with 
programming, they described feeling accountable for the work they : : 

the basic concepts that are being 


had done in the program. 

covered, via the pre-work. Everyone 
In contrast, in one engagement none of the interviewed participants in this cohort did the pre-work. 
mentioned being accountable to their supervisors for how they That's the real potential, maximizing 
implemented what they'd learned during the New Leaders summer ‘ 


workshops. District administrators gave building leaders autonomy for the opportunities for the in-person 


how they led their teams, and most participants did not use the online sessions, with informed role-play 
resources during the school year, in sharp contrast to how participants or other activities.” 
behaved in the other four regions. NEW LEADERS PROGRAM 


; IMPLEMENTATION STAFF MEMBER 
New Leaders staff involved in blended engagements recognize the 


importance of participant accountability, and that participants will 
get out of the program only what they put into it. 


20 Heba, E. D., & Nouby, A. (2008). Effectiveness of a blended e-learning cooperative approach in an Egyptian teacher education programme. 
Computers & Education, 51(3), 988-1006. 
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(2.) Prior Experience With Online Learning 


Studies of blended learners show a positive relationship between “And in the audience we have some 
previous experience with online tools and beliefs about their 
own abilities, or self-efficacy, in learning outcomes.”' Yet, for 
adult learners, age is widely perceived as negatively influencing 


people who are technologically 
savvy. We should decrease stuff on 


success in online and blended learning, due to course designers’ hard paper and increase our use of 
beliefs about technological illiteracy among older adults.” Some technology. But some people prefer 
surveys show that younger professionals with greater exposure old-fashioned paper-and-pencil. 


to online education report less skepticism of blended and online 
learning than their older peers, whereas others suggest that a gap 
in technological experience and mindset is closing among students 
of different ages.?? Conversely, when given a choice of instructional 
modes, older adults are more likely than their younger peers to 
select online learning for its flexibility. Older adult learners are also more likely to seek help 
and engage with peers online, behaviors that are associated with positive outcomes in 
these environments. 


So we need to be malleable.” 


NEW LEADERS PROGRAM IMPLEMENTATION 
STAFF MEMBER 


Interviews conducted by New Leaders researchers indicated that most blended partici- 
pants were experienced online learners. For example, many participants mentioned prior 
experience with online learning in coursework for advanced degrees and professional 
certifications, which helped them know how to plan for adequate time and space to 
engage with the online instructional content. In one engagement, participating principals 
and some of their staff members had prior experience with New Leaders programming; 
they mentioned that their experience with the New Leaders model, vocabulary, and tools 
made it easier to engage with the blended program because they knew what to expect 
and how to implement what they were learning into their practices. 


— 
(3.) Self-Regulation 


Adult students in online classes who report better self-regulation (e.g., setting goals, 
monitoring their understanding, and adapting their thinking) realize greater academic 
achievement.*° Meanwhile, students who earn failing grades in online courses cite a 
lack of study time and absenteeism as primary causes.”° 


Communicating a blended program's daily, weekly, and monthly time requirements 

up front in a clear fashion allows participants to plan accordingly and to thoughtfully 
schedule when they will engage with online content and undertake independent 
assignments. Effective blended programs will facilitate participants’ planning with 
regular communication about the program’s timeline, and with well-placed reminders 
about upcoming deadlines. The established structure and expectations of the Emerging 
Leaders program allowed many participants to plan and schedule their work. 


21 Shea, P and Bidjerano, T (2010). Learning presence: Towards a theory of self-efficacy, self-regulation, and the development of a 
communities of inquiry in online and blended learning environments. Computers & Education, 55, 1721-1731. 


22 Wolfson, N. E., Cavanagh, T. M., & Kraiger, K. (2014). Older adults and technology-based instruction: Optimizing learning outcomes and 
transfer. Academy of Management Learning & Education, 13(1), 26-44. 


23 Betts, L.R., Hill, R., & Gardner, S. E. (2017). "There's not enough knowledge out there”: Examining older adults’ perceptions of digital 
technology use and digital inclusion classes. Journal of Applied Gerontology, October 2017. 


24 Deschacht, N., & Goeman, K. (2015). The effect of blended learning on course persistence and performance of adult learners: A 
difference-in-differences analysis. Computers & Education, 87, 83-89. 


25 Moos, D. C., & Ringdal, A. (2012). Self-regulated learning in the classroom: A literature review on the teacher's role. Education Research 
International, 2012, 1-15. 


26 Mansur, A. B. F., & Yusof, N. (2017). Comparative classification of student's academic failure through social network mining and 
hierarchical clustering. Computer Engineering and Applications, 6(2), 79-86. 
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“| learned early on that | have to put in my calendar a week in advance before pre-work is due to 
make sure that | get it done. So it's definitely something you have to learn, that you can't open up 


on Friday and think you're going to get everything finished.” 


“D," PARTICIPATING TEACHER 


Nevertheless, finding the time to engage with online instructional content and complete 
assignments was a challenge for some participants in New Leaders’ blended programming: 


“| knew | had to do the assignments, [but we] didn’t know it was an ongoing thing. Because of that, 
it was kind of like, ‘Oh, by the way, you're going to a week-long thing in the summer.’ And so, we 
did that. Then it was like, ‘Oh, and you've got these books to read and these assignments.’ That has 


been challenging and really stressful.” 
“P." FLEXIBLE-PROGRAM PARTICIPANT 


Motivation 


While self-regulation creates the conditions for blended learners 

to successfully complete a course, motivation provides the impetus 
to engage in these behaviors.?” Comparisons of student survey data 
and course grades show that motivation corresponds with perfor- 
mance in blended learning courses.®??°° The field of adult learning 
theory often takes for granted that adult learners are self-motivated, 
but the participation in flexible engagements organized by districts 
or other supervisory bodies is often a mandatory part of a given role, 
even if it is presented to participants as optional. In-service teachers 
and administrators may believe in the importance of ongoing profes- 
sional development, but it may not motivate them over long periods 
if the training does not carry over into actual practice.*" 


Across all engagements, program participants often mentioned 
their motivations for engaging with New Leaders’ blended content: 
personal growth, professional growth, and achievement of team 
goals for student outcomes. The virtual and in-person coaching that 
participants received was often cited as a very influential motivator. 


“Il think almost every conversation 

| started with [my coach] was, 
‘Remember, | have no idea what 

I'm doing.’ And she was always very 
understanding about it, and she 
didn't let me make excuses. Even 
though | really wanted to, she didn’t 
let me say, ‘Well, that’s fine.’ She 
definitely pushed me, and | was 
appreciative of that, and being very 
compassionate and understanding 
of how | felt, and especially feeling 
stretched so thinly at times.” 

“L," PARTICIPATING TEACHER 


“Before the training, you know, | just saw things from the perspective of a teacher of what's happening 
in the classroom and why is admin doing this or why are they doing that over there. Now that I’m 
admin and went through the training and seeing everything for myself, | see all the different levels 


and how all the different pieces make a school work.” 
“M," PARTICIPATING ASSISTANT PRINCIPAL 


27 Baird, D. E., & Fisher, M. (2005). Neomillennial user experience design strategies: Utilizing social networking media to support “always on” 


learning styles. Journal of Educational Technology Systems, 34(1), 5-32. 
28 Garrison and Kanuka (2004). 


29 Lim, D.H. & Morris, D.L. (2009), Learner and Instructional Factors Influencing Learning Outcomes within a Blended Learning Environment. 


Educational Technology & Society, 12 (4), 282-293. 


30 So, H.J. & Brush, T.A. (2008). Student perceptions of collaborative learning, social presence and satisfaction in a blended learning 


environment: Relationships and critical factors. Computers & Education, 51 (2008) 318-336 


31 Baker, J., Chaseling, M., Boyd, W., & Shipway, B. (2018). Teachers’ response to a new mandatory professional development process: 


Does it make a difference? Professional Development in Education, 44(4), 570-582. 
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(5.) Relevance 


For teacher leaders and administrators, who dedicate precious work “Honestly, | feel like it all clicked. 
hours to professional development, the issue of relevance is of central [The New Leaders content] all fit in 
importance. In a recent study of in-service teachers who were required to . egade. du Bega ; 
participate in a blended professional development program to improve with our district initiatives, with our 
reflective teaching practice, participants’ perceptions of the program school initiatives. It all made sense.’ 
differed according to their professional ambitions and future goals.* "S," PARTICIPATING PRINCIPAL 


Relevance was highly salient among participants in New Leaders’ 

blended programs: of the five key dimensions of adult blended learning, Relevance 

was the dimension most frequently mentioned by the interviewed participants, and their 
comments specifically focused on positive and negative aspects of the blended content’s 
relevance to their daily practice and future goals. Adult learners in job-embedded 
professional development may be especially attuned to — and better able to assess — 
the relevance of program content to their current work. 


New Leaders staff members know the importance of perceived relevance between the 
program's content and participants’ work priorities and concerns, and they are eager to 
bridge disconnects between participants’ local context and program tools or vocabulary. 


“Oftentimes there are general statements about what's working but nothing specific until you go 
there. Being able to go into the school and work with teams was one of the things they appreciated 
the most and was one of the most beneficial.” 

NEW LEADERS PROGRAM IMPLEMENTATION STAFF MEMBER 


32 Baker, Chaseling, Boyd, & Shipway (2018). 
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Chapter 3 

Serving It Up: 

Blended Discussion 
and Recommendations 


This investigation of New Leaders’ implementation of blended learning programs 

reveals how issues pertaining to Relevance, Accountability, Motivation, Self-Regulation, 
and Prior Experience with online learning shaped participants’ experiences to varying 
degrees. (For further detail, see the case studies in Chapter 4.) Differences between 
engagements in program duration, participant professional role, program structure, and 
content corresponded to differences in the prominence of some themes in participant 
interviews (e.g., prior experience or accountability). Taken together, the five blended 
engagements in this study offer wider insights about developing and delivering 

blended adult professional development programs. 


Relevance Infuses Everything 


Across unique regional cuisines are widely used staple ingredients, like onions or 
butter. Relevance was the one ingredient mentioned by all participants in New Leaders’ 
blended programs. Participants need to see how the program’s content and activities 
relate to, and can be implemented in, their immediate work, in order to find engaging 
with the program worthwhile. The relationship between perceived relevance and overall 
engagement was evident in differences among the programs in this study. 


Relevance Recommendations: 


1. Aid participants in integrating course content into existing school or district-based professional 
development structures. 


2. Provide adequate planning time to ensure that staff have knowledge of local assessment rubrics and 
teaching/instructional leadership standards before launching the program. 


3. Identify points in a program's content that may require explicit translations to local context. 


Accountability Can Enhance Self-Regulation 


Much like salt is difficult to detect in a fully prepared meal, accountability structures 
often go largely unnoticed by participants during programming, even as they enhance 
the experience. Most of the participants we interviewed did not appear to be cognizant of 
how the blended programs’ structures (e.g., formal assessments, nudges from directors, 
or group presentations) were designed to assist them in getting the work done. But 
program designers are acutely aware of the need for effective accountability structures 
to ensure completion by participants who are beset with seemingly countless demands 
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for their time and attention. In their interviews with researchers, New Leaders staff 
frequently mentioned the importance of participant accountability — New Leaders’ 
blended programs require active learning and focused engagement with instructional 
content and assignments. 


If a meal is prepared with little or no salt, diners may need to add their own. One of 

the participating districts in this study revealed the importance of accountability as an 
ingredient in a blended professional development program. Judging that school leaders 
could best decide how to use their time, the New Leaders staff and the school district 
designed a program whereby principals and their staff could opt in to program elements 
they deemed to be useful. The participants we interviewed in the district emphasized 
the relevance of the content presented during in-person training, as well as the potential 
appeal of the virtual coaching that was made available to them, even as they confided 
that they had not made much use of either offering. Without accountability, program 
engagement was low. 


Interviews with participants in all regions reveal the extent to which accountability and 
self-regulation are interdependent. New Leaders’ programs are, by design, rigorous and 
robust; they demand time and attention for proper implementation. When the majority 
of content is accessed through an online learning portal, participants must practice 
self-regulatory behaviors, such as setting aside dedicated time to watch instructional 
videos, taking notes, or engaging in reflection. Some accountability strategies can 
scaffold participants’ self-regulation strategies, encouraging them to consciously assume 
control of their experience in the program — for example, with explicit communications 
about how and when participants will be expected to do independent work (i.e., during 
the school day or after-hours) and with up-front summaries of how long each online/ 
independent component typically takes. 


While a lack of salt can result in a bland meal, too much salt can make food unpalatable. 
Participants in blended programming for professional development may consider highly 
structured accountability mechanisms to be heavy-handed and off-putting, rather than 
helpful. Blended-program designers and facilitators can take advantage of lighter-touch 
accountability mechanisms: For example, cultivating interpersonal engagement during 
in-person group sessions can serve both as motivating factors and as accountability 
mechanisms. Blended programs were structured so that the in-person sessions (which 
participants value highly) built upon independently completed pre-work, prompting 
participants to prioritize completing that pre-work. Some participants divulged that they 
watched videos multiple times, took notes, and prepared specifically for group discus- 
sions. In short, they did what was necessary to be ready to make valuable contributions 
to group sessions with their peers. 


Accountability & Self-Regulation Recommendations: 


1. Come to an explicit agreement, at the outset, with school/district partners about when participants will 
be expected to do independent work, thereby allowing the participants to anticipate the program's time 
demands and to plan accordingly. 


2. Support participants in developing habits of self-regulation, by highlighting video durations and estimated 
time that modules or assignments will require. 


3. Use technology to support accountability and personal interdependence (e.g., nudges through text, status 
monitoring of participants’ and/or the group's progress). 


4. Explicitly set expectations for online work to be done in advance of in-person sessions, which focus on peer 
discussion and application of concepts. 
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When Motivations Are Met, Satisfaction Ensues 


Myriad reasons can inform the choice of what, if anything, to cook — e.g., personal 
preferences, a special occasion, what's in the refrigerator, or a need to eat more vegetables. 
The relevance of New Leaders’ programming to professional work was uniformly the 
strongest driver of participant engagement. But interviewees also spoke at length about 
their satisfaction with different aspects of the program — e.g., their coaches, cohorts, 

and the quality of the program overall — revealing other factors that motivated them to 
participate (or made participation more difficult). The motivating catalysts and dampeners 
ranged widely, but participants repeatedly mentioned a few specific motivators. 


Across sites, participants described the value of interacting with their coaches and 
facilitators — to ask questions, and to dig deeper into content they'd learned during 
independent work. For some participants, a relationship with a New Leaders coach was 

a unique way to receive feedback from an experienced school leader who was not respon- 
sible for evaluating their performance. Participants appreciated that coaches possessed 
substantial professional experience and were interested in their unique school placement 
and responsibilities. 


Participants also valued the opportunities that New Leaders’ programming created for 
them to encounter and network with peers in other buildings (or districts), with whom 
they otherwise did not typically interact. Some participants with years of professional 
experience in their district expressed appreciation that the New Leaders program 
afforded them valuable opportunities to interact with peers, opportunities that they 
otherwise did not have. 


Finally, when individual participants are in professional transitions — e.g., working ina 
new position, or with a new team — they are more open to experimenting and adopting 
new processes and resources. 


Motivation, relevance, and self-regulation can be interdependent. Participants who 

viewed the program as a necessary professional development training step, on a path 
toward expanded leadership roles and/or promotions, were motivated not only to com- 
plete assignments and independent modules, but also to show their best work. Conversely, 
in one site, participants learned that they would not be able to move from their current, 
first-year teacher-leader training into a second-year resident principal training course, 

due to fiscal constraints; participants reported their disappointment, and their program 
director observed a corresponding decrease in timely assignment submissions. 


By design, blended programming directs participants to online platforms and tools. 
Blended participants’ experience is inextricably tied to the design and functionality of 

the technology they are asked to use. No matter how relevant program content might 

be, when participants encounter technical difficulties that cannot be remedied easily and 
swiftly, their motivation to complete online pre-work or submit assignments can suffer. 
Technology problems — e.g., assignments that took a long time to upload to the online 
platform, or failures of videoconferencing software that disrupted scheduled virtual coach- 
ing sessions — were commonly identified as motivation dampeners. For participants who 
completed these assignments at night, after work, such glitches were especially burdensome. 


Motivation Recommendations: 
1. Provide real-time, in-house tech support for the virtual platform, including evenings and weekends if possible. 


2. Develop technology to allow program participants to engage across districts, in order to share ideas and 
exemplars and build professional networks. 
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Prior Experience 


Making a meal is easier when its ingredients and preparation techniques have been used 
before. While most participants interviewed for this study had no prior experience with 
New Leaders, those who did, or who worked with supervisors or colleagues who had prior 
experience with New Leaders, more readily recognized the relevance of the program's 
tools and content to their work. 


All participants we interviewed reported some experience with online learning, such as in 
graduate school or to fulfill licensing-renewal requirements. Even veteran teachers who 
had been working for several decades did not describe themselves as online novices. In 
fact, some were concurrently engaged in online learning through their districts as part of 
other professional development initiatives. Although many participants shared that they 
preferred the in-person sessions to independent online work, no participants reported 
being unable to access or use New Leaders’ online resources. 


Participants’ familiarity with online instructional sites ranging from online university 
courses to Khan Academy tutorial videos makes them savvy connoisseurs of website 
and video design. They were quick to offer suggestions about how to increase online 
engagement (e.g., with online discussions and interactive instructional modules) and 
how to improve the online platform used in the 2017-18 school year. 


Prior Experience Recommendations: 


1. Recognize that many, if not most, blended-learning participants have prior experience with online learning. 
It's imperative that web platform and online content designs reflect state-of-the-art standards that are 
regularly upgraded and feature routinely updated content. 


2. Introduce the online platform as early as possible, preferably before the first in-person meeting, and then 
address questions or difficulties if they arise, rather than devoting precious in-person time to the rudiments 
of the online platform. 


Different Recipes Yield Different Outcomes 


While New Leaders’ established program models are standardized in content, structure, 
and designated outcomes, each of the five blended engagements in this study was unique 
in its design and content. To provide further detail about how Relevance, Motivation, 
Accountability, Self-Regulation, and Prior Experience inform blended participant experi- 
ences, the next chapter presents a sample of three case studies of New Leaders’ blended 
programming in the 2017-18 school year. 
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Chapter 4 


From Scratcn: 
Three Blended Case Studies 


Case Study 1 
Coming Back for Seconds: 
Learning Through Experience 


“Just having a year under our belts is a little easier to navigate this year.” 
“E,"” PARTICIPATING PRINCIPAL 


New Leaders has a long history of partnership with this large, urban district. For the 2017-18 
school year, the district reported that its student population was 79 percent African Amer- 
ican, 10 percent Hispanic/Latino, 8 percent White, and 3 percent Other race/ethnicity. In 
the 2016-17 school year New Leaders had delivered the entirely online Transforming Teams 
program to four schools in the district. In 2017-18, two of the school principals who had been 
in the Transforming Teams program joined another New Leaders veteran principal in a new, 
blended version of Transforming Teams, whereby they and their respective instructional 
leadership teams engaged with online content combined with in-person group sessions 

and individual virtual coaching with New Leaders staff. 


Each principal in this cohort selected a series of modules from New Leaders’ Transforming 
Teams curriculum for their Instructional Leadership Teams (ILTs) to complete. Each module, 
designed to take three months to complete, consisted of tutorials followed by practice, 
implementation, and monitoring before ILTs met with a virtual advisor for feedback. 


This engagement included in-person touchpoints, such as a “program launch” that consisted 
of an in-person district kickoff, a 60- to 90-minute introductory phone call with a virtual 
advisor, and activities led by the school principal. It also involved ongoing support from 

a New Leaders’ program advisor, including phone calls during the school year offering 
guidance and feedback; participation in mid-year and end-of-year “step-backs” (formal, 
structured reviews of program data about progress to goals, providing opportunities to adjust 
when needed and to discuss long-term implications for implementation) with ILTs engaged 

in the program; and an in-person visit to each school, for additional support. 


Lastly, at the end of the year, a day-long program symposium sponsored by a local funder 
was held at one of the participating schools. Facilitated by the schools’ New Leaders advisor 
and other New Leaders program staff, attending principals, assistant principals, and selected 
ILT members engaged in a series of focused discussions and activities that expanded on their 
work during the year with the Transforming Teams modules. 
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This Blend Pairs With: Self-directed, heavily virtual programs work best with 
participants who have prior experience with New Leaders’ programs, and who are eager 
to continue their learning and access the resources New Leaders provides. 


Result: 


Engaged, confident, and satisfied school leaders quickly integrated their learning into their daily practice and led 
teacher teams through training modules. Participants requested ongoing access to the New Leaders resources 
to continue change-management initiatives. 


Special Sauce for a Successful Engagement With Experienced 
Alumni: Relevance, Experience, Motivation, and Accountability 


As in other engagements in this study, relevance was the most prevalent theme in 
interviews with these participants. The perceived relevance of the program to the 
participants’ personal goals, professional goals, the realities of their jobs, and the 
expectations of their supervisors and districts strongly informed their experience 
in the program. The three principals had prior experience with New Leaders and 
strongly believed in its relevance and efficacy. Some ILT members 

initially doubted the relevance of some of the New Leaders content 


to their own roles, but with guidance from their principals, they “So this course...| started into it 
eventually came to appreciate its applicability to their work. and | was like, ‘Oh this doesn’t 
All of the principals and many of the participating ILT members apply to me, this is for [teachers 
had prior experience with online learning. The three principals of] older kids.’ But guess what? | 
in the study each had participated in New Leaders’ programming found out that there was a lot of 


in previous years; two of the three had previously completed the 
Transforming Teams program. However, most of the ILT members . 7 
were participating in New Leaders’ programming for the first in my [preschool] classroom. 
time in school year 2017-18. “N," PARTICIPATING ILT MEMBER 


valuable material...that | could use 


“New Leaders has been extremely good to me as a person and as a principal. Through New Leaders 
| started to take an honest look at who | am as a person. When you become a better leader, you 
become a better person in my opinion, and so it has not failed me.” 

“B," PARTICIPATING PRINCIPAL 


Regular check-ins with their virtual advisor motivated all participants, and some 
expressed an interest for greater flexibility in the types of interactions available to 
them. Two of three principals had an established relationship with their advisor from 
the previous year and mentioned the corresponding trust that encouraged them to 
keep their teams engaged with the program, even in the face of other urgent priorities. 
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The principals’ understanding of the expectations and payoffs of the New Leaders’ 
program compelled them to set clear expectations with their teams at the beginning of 
the year about the program's pace and the commitment necessary to benefit from the 
content. The principals developed strategies to involve ILT members in the program by 
incorporating it into existing professional development structures. 


“The teachers on my ILT, none of them had ever been on an ILT before. So there's a lot of transition... 
and that means that everybody's growing. So everybody's going to be in an uncomfortable space.” 
“V" PARTICIPATING PRINCIPAL 


Watch Out For (Potential Engagement Dampener) 


Relevance: |ILT members described some challenges in implementing what they learned 
in New Leaders modules, as they encountered resistance from some of the teachers with 
whom they work. The challenge of persuading colleagues to embrace changes in practice 
is both common and not unique to blended professional development models. 


“| don't know if it was a pattern, but [my peers] that tend to be more timid with the technology are also 
the ones that are more timid with analyzing data and looking at data. My grade-level partner is scared 
to death of technology, but she also thinks, ‘l know my kids. | don’t necessarily need the numbers.” 

“E," PARTICIPATING ILT MEMBER 
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Case Study 2 
Into the Pressure Cooker: 
Fast-Paced Learning 


“LIt's been] very beneficial. I'm a little nervous that...all the modules will be disappearing at the end 
of the school year because | would love to have that information longer for us to continue to utilize, 
because it’s some very valuable information on there.” 

“E," PARTICIPATING PRINCIPAL 


This county-wide district has a student population that is 55 percent White, 19 percent Black, 

17 percent Latino, 5 percent Asian, and 4 percent Other race/ethnicity. New Leaders’ relationship 
with this district began in January of the 2017-18 school year, with the launch of a blended 
program for three elementary school principals and their instructional leadership teams, 

in support of the district's strategic “Autonomy” initiative, whose purpose is to give building 
leaders more independence and decision-making authority in support of student achievement. 
The district chose school leaders who met criteria for success in the district's autonomy 
initiative to participate in the New Leaders’ program. In turn, teams of administrators and 
teacher-leaders were selected by their principals to participate for each school. 


Between January and May 2018, New Leaders facilitators led group sessions focused on topics 
covered in online modules that participants had completed independently prior to convening. 
Three online modules were provided to school teams, to be completed and implemented 
according to participants’ schedules. In-person cohort sessions occurred at the beginning, 
middle, and end of the engagement. A virtual advisor conducted phone calls or video confer- 
ences with participant teams during the program cycle, offering guidance and feedback. 


This Blend Pairs With: Introductory blended engagements that launch mid-year. 


Result: 


School and teacher leaders adopted resources from the High-Performing Teams module to improve 
collaboration and efficiency in their schools. Recognizing the success with implementing these resources, 
participants requested ongoing access to the New Leaders resources to continue learning and help them 
launch the next school year. 


Special Sauce for a Successful Engagement With New Leadership 
Teams: Relevance, Motivation, and Self-Regulation 


Participants reported that the majority of materials presented in the modules were relevant 
to their school’s needs and to their professional practice. In particular, participants found the 
prepared resources, such as leadership rubrics and meeting protocols contained within the 
modules, to be especially useful, and quick and easy to implement during their workday. 


1 


‘Just look at...the protocols and how to bring pieces into place. ... It structures things and gives it — 
makes sure that everybody has a voice and uses that voice. So | think it leaves nobody out. 
Everybody's included; everybody feels valued. So there's a lot of positive in that.” 

“O," PARTICIPATING TEACHER 
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The in-person sessions motivated participants who relished the opportunity to collaborate 
with peers. Conversations among peers experiencing similar challenges helped to reinforce 
the program's relevance. Participants also found the content and coaching to be of high 
quality throughout, which encouraged them to complete instructional modules. Nearly 
all participants expressed a desire to continue the program during the next school year. 


To combat communication issues and the short timeframe, some school-based teams 

“self-blended” online content by watching and discussing videos together during the school 
day. They found this to be an effective strategy for staying on track in advance of the larger 
in-person sessions, as well as for connecting the materials more directly to their school 
context. Teachers appreciated — and were motivated by — administrators who made time 
available to them during the workday to complete the program by providing coverage in 
their classrooms. 


“We did it together online. So we watched many of the videos online together, so we were able to 
stop them and talk about it. And it was better to do it before the presentation because then you knew 
what they were talking about if you already had that experience. And one or two of them, | know the 
one whole section we watched at home by ourselves. | was texting the other teacher...because I'm 
like I'm missing something. You play it over and over because you don't have that help from the other 
people talking about it, so it was good to do it together and then have a meeting face-to-face.” 

“R," PARTICIPATING TEACHER 


Watch Out For (Potential Engagement Dampener) 


Self-Regulation: Initially, participants were confused about the “Initially | didn't get a whole lot of 
exact purpose of the program and were not sure how they were 


supposed to engage with the independent content, which hampered information [about the New Leaders 


their ability to plan and assume control of their experience. Many program]. New Leaders talked about 
interviewees shared that they had very little knowledge of what the this when we first met. Even like the 
New Leaders program was about when it launched. They also found first couple of meetings, | almost felt 


the program's overarching purpose, launch timeline, expectations, 
and anticipated outcomes to be confusing. In this context, the 
subsequent virtual coaching was vital for orienting teams to the 
program's materials and expectations. “M," PARTICIPATING PRINCIPAL 


like | was coming late to a party. ... It 
was nice to see it work through itself. 


" 


Participants also reported resistance from other staff when they 

tried to implement what they had learned. Although staff resistance to new tools and 
processes was reported in other districts, the pace of this half-year program — and the 
speed of implementation — may have exacerbated these issues by failing to account for 
the time needed to garner support for these change-management efforts. 
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Case Study 3 
Omitting the Optional: 
Self-Directed Learning 


“New Leaders really made a point of sitting down with our senior leaders to really say, ‘What is 
working? What is not?’ I'm sometimes floored at the way other partners show up and come in with 
a different attitude, whereas New Leaders says, ‘How can we get better at this and what can we do 
differently?’ | do really appreciate that.” 

“E,” SENIOR DISTRICT ADMINISTRATOR? 


In the 2017-18 school year New Leaders began working with a large district that adheres 
to a policy of autonomy, giving school leaders latitude to choose resources for their teams, 
and to decide how to use their chosen resources. As a result, this flexible engagement for 
instructional leadership teams was unique among the five study sites for its deliberate 
lack of participant accountability. The district made the program available to its schools 
as a part of a menu of professional development options. Participants who opted in to the 
New Leaders program attended a mandatory three-day in-person training session before 
the school year began, and then were provided access to the online Transforming Teams 
platform and a virtual coach for ongoing support. However, by design, all engagement 
with New Leaders resources beyond the in-person training was optional for participants. 


Twenty-five district schools participated in the 2017-18 New Leaders training. New 
Leaders facilitators delivered selections of Transforming Teams content to ILT members 
over the course of three days in August 2017. Instructional Superintendents also engaged 
in ongoing professional development with New Leaders, and they were present for the 
training of ILT members. During the in-person August training, facilitators introduced 
participants to the online modular content that New Leaders made available to them 
throughout the year, and to the coach who was available for consultation “office hours” 
via phone. 


This Blend Pairs With: Districts that offer high-quality resources to high- 
autonomy school leaders. 


Result: 


Although participants reported recognizing the value of the content they learned during the summer in-person 
training sessions, very few of the ILT participants used New Leaders’ online resources during the 2017-18 school 
year, and only one principal took advantage of the offered virtual coaching. 


33 Interviews with district central office administrators were not coded for thematic content, since they did not participate in the New Leaders program, 
but their input is included to provide context about the district's expectations and communications with its school staffs and with New Leaders. 
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Special Sauce for Flexible Participants With Optional Resources: 
Relevance and Motivation 


As in other engagements in this study, relevance was the most prevalent theme in interviews 
with participants; two district administrators we interviewed also mentioned the importance 
of the relevance of the New Leaders’ content to school leaders’ work. A program participant 
who was newly appointed to a school leadership role shared this observation: 


“The training definitely prepared me for this school year, and almost everything that we discussed 
and went over, we pretty much implemented here, and it runs very smooth and | love the way that 
things are working out since we've had the training.” 

“M," PARTICIPATING TEACHER 


Another participant emphasized the guidance New Leaders provided in getting an 
instructional leadership team up and running, a priority for both the school and the district: 


“The ILT functions as an extension of the principal. New Leaders gave us the instructional team vision.” 
“B," PARTICIPATING ASSISTANT PRINCIPAL 


Interviewees connected their receptiveness to using New Leaders resources to the fact 

that they were in transitional junctures — being new to a school, new to a leadership role, 
and/or having anew team unfamiliar with the principles of instructional leadership. For these 
participants, the 2017-18 school year brought new challenges for which they did not yet have 
established solutions or processes. They were actively seeking tools and support and, as a 
result, were open to incorporating New Leaders’ resources into their practices. 


One principal took advantage of the opportunity to get one-on-one coaching once the school 
year was underway, motivated by both her need and her perception of the New Leaders 
coach's expertise and shared perspective: 


“ 


| was in that space of, ‘Good gosh, I'm worn out.’ This is not something | would be able to sustain 
until forever. So, | was like, ‘Maybe I'm doing something...maybe I've been out of the game for a 
while, maybe I’m thinking about this wrong.’ What the heck, they just sent this invite, am | going to 
talk to these guys? They're not judge-y. ...So, we jumped on the line and | said, ‘Look, here's where 
I'm at, here’s what I'm thinking. | don't have this super formal protocol for you. I'm not even going 
to pretend I've done this or that.’ We just chatted and, really, | think it was just that affirmation that 
| was like, ‘Okay, you might be crazy for jumping back in the game, but you haven't lost your mind 
in terms of the steps you're trying to make.” 

“R,” PARTICIPATING PRINCIPAL 
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Watch Out For (Potential Engagement Dampeners) 


Accountability: As mentioned in Chapter 3, participant accountability in this district was 
entirely absent, by design. The district's intentions in making access to New Leaders’ online 
content part of the flexible engagement were unclear to the participants we interviewed. 
One principal observed: 


“If it really is just, ‘Here's a resource,’ so be it. | don’t think that’s a terrible thing, but just name that 
and be okay with that. Or if really, here’s the outcome we expect, then | think we have to be really 
intentional with who we're connecting to the program, so that those outcomes can be drawn.” 

“R," PARTICIPATING PRINCIPAL 


Self-Regulation: When resource use is optional, making the time to “| think that there wasn’t much 

access online content may not be prioritized, even when participants throughout the year connection. So 

recognize the value and relevance to their work. Ge suil Sienite fa ‘ 
not to say it didn't sit in my head, but 


Our interviews with participants also highlighted the self-regulation my reminder would be when | would 


and attention challenges participants can face when they are pre- get that, ‘Hey, they're available for 
sented a lot of information during intensive, multi-day sessions, 


virtual office hours’ email. Short of 
as in the August 2017 kickoff session. 


that, there weren't other connections 
where folks were saying, ‘Hey, what 
about this [online module]? or, ‘Did 
you remember that [tool]?’”” 

“B," PARTICIPATING ASSISTANT PRINCIPAL 
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Chapter 5 


Conclusion 


Successful blended adult professional development programs don't stem from a single 
recipe, as illustrated by the case studies in this report. As New Leaders continues its 
expansion of blended programming, it has a growing collection of blended models to ref- 
erence, for what works and what to watch out for, in a variety of programmatic contexts. 
While there isn't a single master recipe for blended programs, the analyses conducted 

for this evaluation have revealed how issues pertaining to Relevance, Accountability, 
Self-Regulation, Motivation, and Prior Experience shaped the experiences of participants 
across engagements. These five important ingredients must be considered when scoping, 
building, and delivering blended adult instruction. 


This study also revealed the hidden challenges in the organization's evolution from 
traditional, in-person instruction to blended models that incorporate independent online 
work and virtual coaching with the in-person sessions and job-embedded practice 

that have been the hallmarks of New Leaders’ successful historic programs. Blended 
instructional models are not interchangeable with traditional, in-person models; they 
require adequate organizational investments in relevant technology, and in staff time to 
develop, prepare for, and support the effective delivery of both the online and in-person 
instructional components. 


When a blended instructional recipe is carefully adapted to suit the local context, 
the results speak for themselves. 


“| feel like any time [New Leaders coaches] have been involved with us has felt really inspiring, and | 
feel like they've been helpful in facilitating our conversation to kind of have the outcome that would 
be the best for our school. So | really feel positive after even the face-to-face or the tutorials, the 
Skype [meetings], you know, all those pieces, it kind of rejuvenates my motivation.” 

“H," PARTICIPATING ASSISTANT PRINCIPAL 
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Appendix A 
Participant Program Survey Responses 


Across engagements, participants widely reported satisfaction 
with both in-person and online program segments. 


Engagement A (described in “Omitting the Optional”): 
Participant Post-Training Survey Feedback 


In-Person Sessions 


Session content was of high quality. 93% 
Overall, my knowledge in the content areas covered during these sessions has increased. 100% 
The sessions have improved my ability to create the conditions in my school to improve 93% 


student achievement. 


Engagement B (described in “Into the Pressure Cooker”): Participant Survey Feedback 


Online Program Components 


Online content was high quality. 


100% 


Online content was relevant to our work. 


Overall program alignment 


78% 


| will be able to use what | learned in this session in my professional environment. 100% 
The program has improved my knowledge. 100% 
Session content aligned with my district's goals. 100% 


Engagement C (described in “Coming Back for Seconds”): Participant Survey Feedback 


Online Program Components 


Online content was high quality. 


97% 


Online content was relevant to our work. 


97% 


Virtual Coaching 
The support provided during the office hours supported clarity and action steps. 


In-Person Symposium 


100% 


| will be able to use what | learned in this session in my professional environment. 94% 
Overall, my knowledge in the content areas covered during the session has increased. 94% 
Session content aligned with my organization's strategic priorities. 100% 


Ca 
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Engagement D: Participant Survey Feedback 


Online Program Components 


Content was high quality. 


100% 


Content was relevant to my work. 


100% 


Coaching 
The support provided supported my growth. 
Virtual Platform 


100% 


Technology was user-friendly. 83% 
The modules provided the necessary background information for sessions. 100% 
The platform helped me learn the content. 100% 


Engagement E: Participant Survey Feedback 


Online Program Components 


Online content was high quality. 


100% 


Online content was relevant to my work. 


In Person Cohort Sessions 


92% 


My Emerging Leaders peers provided feedback that was helpful in building my 
leadership capacity. 


One-on-One Coaching 


My director provided coaching that helped me develop a SMART goal aligned to my 
students’ growth areas. 


Online Platform 


100% 


94% 


The online platform helped me learn the content. 


Ca 
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Appendix B 
Measuring the Key Ingredients in Participants’ Experiences 


Catalysts or Dampeners: The research team identified mentions of the five key ingre- 
dients of adult blended instruction described above in participant interviews. Each of 
these five dimensions of blended instruction can occur as a “catalyst” — which supports 
participant engagement with the program — or as a “dampener,” which impedes partici- 
pant engagement. 


Forty program participants were interviewed individually or in small groups. Thirty-six 
interview transcripts contained 1,337 “quotations” (discrete statements related to their 
work and experiences in the New Leaders program), which were coded for engagement 
themes related to Accountability, Prior Experience, Motivation, Self-Regulation, and 
Relevance. 


PROGRAM CATALYSTS AND DAMPENERS 
35% 
29% 
30% 


25% 


@ Catalyst 
@ Dampener 


21% 
20% 
15% 12% B% 
10% 8% 7% 
9 
5% A% 5% 
1% 1% FF 
0% = ol | | 
x @ of sof @ 
a ro “ <0 ah? oo 
ee ok Nhe gar ee 
i 


Across engagements, 67 percent of the coded quotes pertained to engagement catalysts, 
of which those related to Relevance (29 percent) and Motivation (21 percent) were most 
frequently mentioned. Relevance also was the most frequently mentioned engagement 
dampener (13 percent of all quotations), closely followed by Self-Regulation engagement 
dampeners (12 percent). 
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Appendix C 
Participant Interview Protocol 


Researchers used this interview protocol to guide semi-structured interviews with participants 
in New Leaders’ programs. Before the interview, interviewees signed an informed consent form 
to communicate the goals of the study, its risks and rewards, and the protection of identifiable 
information. Researchers recorded interviews and took handwritten notes during the interviews. 


Opening 


Establish Rapport: Hello, mynameis_ andi’mthe__- 


at New Leaders. It's a pleasure to meet you. Thank you so much for making the time to 
speak with me today. 


Purpose: We're conducting a study with the support of the Bill & Melinda Gates 
Foundation to figure out how to improve our blended learning programs- — leadership 
development training that mixes in-person and online learning. 


Motivation: |'m interested in learning from your experience participating in___ . | want 
to emphasize that I’m not evaluating your participation; I'm talking to participants like 

yourself all over the country to see what people are saying about these programs. And as 
a thank you, we can provide you with a $500 gift card to DonorsChoose.Org to purchase 


supplies and resources for your classroom or school. 


Consent: Does that sound good to you? Okay, here's a copy of the consent form. Take a 
moment to read it through, and if you'd like to proceed, sign at the bottom. Then we can 
get started. And if it’s okay, I'd like to record our conversation so that | can go over the 
details later. 


Time Line: | know that your time is valuable. This interview should just take about 30 
minutes. Does this still fit in your schedule? 


Transition: Great! Let me begin by asking you a few questions about your school. 


Context Script 
- So, it’s early on Monday morning and students are arriving. What do you think | would notice coming in through 
Question 1 
the door of the school? 
Question 2 And if | visited here a year ago, what would be different from today? 
s So now I'd like to shift to your participation with New Leaders. How would you describe your experience with the 
Question 3 
program overall? 
Question 4 What have you learned in the program? What types of support did you receive? 
Question 5 What are your most important professional goals for your school this year? 
Question 6 How would you describe the objective of the program from New Leaders? 
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Overall, what did you think of the in-person component? Have you implemented any of the strategies that you 


Question 7 learned? 

Question 8 Do you have any suggestions to improve the in-person content of the program? 

Question 9 How did the in-person sessions prepare you for the online and virtual components of the program? 

Question 10 Overall, what did you think of the online elements? 

Quection ti Did you experience any challenges in completing the online segments of the program — e.g., problems accessing 
the platform itself, or making time to sit down with the computer, or something else? 

Question 12 Were there any parts of the online content that might have been better delivered in-person? 

Question 13 Returning to the program overall, were there any strategies or practices that you found difficult to implement, or 
think that you would have difficulty implementing if you tried? 

Question 14 And are there any practices or strategies that were very easy to implement? 

Question 15 Do you have any feedback for how any aspect of the program could be improved? 

Closing 


Summary: [Summarize the major points raised in one to two sentences. ] 


Maintain Rapport: | appreciate your taking the time to talk with me today. Your insights 
have been very illuminating. Is there anything else that you can think of that would be 
helpful for me to know? 


Action Steps: I'll be bringing this information back to New Leaders to help guide the 
creation of new tools and resources. Is it alright if | follow up with you if | have any 
questions? In the next few weeks, you'll receive an email with the $500 gift card to 
DonorsChoose.Org. Please let me know if you don’t receive it. Thanks again. 
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